Achievement, learning, and seasonal impact as measures of school effectiveness: it’s better to be valid than reliable

When evaluating schools’ effectiveness, is it better to use absolute achievement levels or to use learning rates over a 9or 12-month period? Or is it better to use a new measure, seasonal impact, which is defined as the acceleration in learning rates that occurs when students finish summer vacation and start the school year? Answering this question involves a tradeoff between validity and reliability, since unfortunately the most reliable measure (achievement) is the least-valid measure as well. In this paper, we evaluate the tradeoff using a simple covariance-structure model. The results suggest that, when achievement is compared to learning and impact, reliability is a minor issue compared to validity. Although achievement is more reliable than learning or impact, achievement’s advantage in reliability is much smaller than its disadvantage in validity. Our findings support the view that reliability is not a sufficient reason to evaluate schools using achievement levels.